Synthetical Test Data With Python
Definition of Synthetical Data
There is hardly any engineer or scientist who doesn't understand the need for synthetical data, also called synthetic data. But some may have asked themselves what do we understand by synthetical test data? There are lots of situtations, where a scientist or an engineer needs learn or test data, but it is hard or impossible to get real data, i.e. a sample from a population obtained by measurement. The task or challenge of creating synthetical data consists in producing data which resembles or comes quite close to the intended "real life" data. Python is an ideal language for easily producing such data, because it has powerful numerical and linguistic functionalities.
Synthetic data are also necessary to satisfy specific needs or certain conditions that may not be found in the "real life" data. Another use case of synthetical data is to protect privacy of the data needed.
In our previous chapter "Python, Numpy and Probability", we have written some functions, which we will need in the following:
- find_interval
- weighted_choice
- cartesian_choice
- weighted_cartesian_choice
- weighted_sample
You should be familiar with the way of working of these functions.
We saved the functions in a module with the name bk_random.
Definition of the Scope of Synthetic Data Creation
We want to provide solutions to the following task:
We have n finite sets containing data of various types:
D1, D2, ... Dn
The sets Di are the data sets from which we want to deduce our synthetical data.
In the actual implementation, the sets will be tuples or lists for practical reasons.
The process of creating synthetic data can be defined by two functions "synthesizer" and "synthesize". Usually, the word synthesizer is used for a computerized electronic device which produces sound. Our synthesizer produces strings or alternatively tuples with data, as we will see later.
The function synthesizer creates the function synthesize:
synthesize = synthesizer( (D1, D2, ... Dn) )
The function synthesize, - which may also be a generator like in our implementation, - takes no arguments and the result of a function call sythesize() will be
- a list or a tuple t = (d1, d2, ... dn) where di is drawn at random from Di
- or a string which contains the elements str(d1), str(d2), ... str(dn) where di is also drawn at random from Di
Let us start with a simple example. We have a list of firstnames and a list of surnames. We want to hire employees for an institute or company. Of course, it will be a lot easier in our synthetical Python environment to find and hire specialsts than in real life. The function "cartesian_choice" from the bk_random module and the concatenation of the randomly drawn firstnames and surnames is all it takes.
import bk_random firstnames = ["John", "Eve", "Jane", "Paul", "Frank", "Laura", "Robert", "Kathrin", "Roger", "Simone", "Bernard", "Sarah", "Yvonne"] surnames = ["Singer", "Miles", "Moore", "Looper", "Rampman", "Chopman", "Smiley", "Bychan", "Smith", "Baker", "Miller", "Cook"] number_of_specialists = 15 employees = set() while len(employees) < number_of_specialists: employee = bk_random.cartesian_choice(firstnames, surnames) employees.add(" ".join(employee)) print(employees)The Python code above returned the following:
{'Simone Looper', 'Laura Miles', 'Kathrin Miles', 'Paul Miles', 'Paul Rampman', 'Laura Singer', 'Laura Chopman', 'John Baker', 'Bernard Chopman', 'Yvonne Bychan', 'Paul Smiley', 'Frank Smith', 'Sarah Smith', 'Frank Chopman', 'Laura Smiley'}
This was easy enough, but we want to do it now in a more structured way, using the synthesizer approach we mentioned before. The code for the case in which the parameter "weights" is not None is still missing in the following implementation:
import bk_random firstnames = ["John", "Eve", "Jane", "Paul", "Frank", "Laura", "Robert", "Kathrin", "Roger", "Simone", "Bernard", "Sarah", "Yvonne"] surnames = ["Singer", "Miles", "Moore", "Looper", "Rampman", "Chopman", "Smiley", "Bychan", "Smith", "Baker", "Miller", "Cook"] def synthesizer( data, weights=None, format_func=None, repeats=True): """ data is a tuple or list of lists or tuples containing the data weights is a list or tuple of lists or tuples with the corresponding weights of the data lists or tuples format_func is a reference to a function which defines how a random result of the creator function will be formated. If None, "creator" will return the list "res". If repeats is set to True, the results of helper will not be unique """ if not repeats: memory = set() def synthesize(): while True: res = bk_random.cartesian_choice(*data) if not repeats: sres = str(res) while sres in memory: res = bk_random.cartesian_choice(*data) sres = str(res) memory.add(sres) if format_func: yield format_func(res) else: yield res return synthesize recruit_employee = synthesizer( (firstnames, surnames), format_func=lambda x: " ".join(x), repeats=False) employee = recruit_employee() for _ in range(15): print(next(employee))The above code returned the following result:
Frank Looper Roger Cook John Smith Bernard Chopman Roger Looper Eve Singer Frank Smiley Jane Cook Laura Singer John Smiley Laura Miller Laura Rampman Jane Singer Yvonne Rampman Sarah Rampman
Every name, i.e first name and last name, had the same likehood to be drawn in the previous example. This is not very realistic, because we will expect in countries like the US or England names like Smith and Miller to occur more often than names like Rampman or Bychan. We will extend our synthesizer function with additional code for the "weighted" case, i.e. weights is not None. If weights are given, we will have to use the function weighted_cartesian_choice from the bk_random module. If "weights" is set to None, we will have to call the function cartesian_choice. We put this decision into a different subfunction of synthesizer to keep the function synthesize clearer.
We do not want to fiddle around with probabilites between 0 and 1 in defining the weights, so we take the detour with integer, which we normalize afterwards.
from bk_random import cartesian_choice, weighted_cartesian_choice weighted_firstnames = [ ("John", 80), ("Eve", 70), ("Jane", 2), ("Paul", 8), ("Frank", 20), ("Laura", 6), ("Robert", 17), ("Zoe", 3), ("Roger", 8), ("Simone", 9), ("Bernard", 8), ("Sarah", 7), ("Yvonne", 11), ("Bill", 12), ("Bernd", 10)] weighted_surnames = [('Singer', 2), ('Miles', 2), ('Moore', 5), ('Looper', 1), ('Rampman', 1), ('Chopman', 1), ('Smiley', 1), ('Bychan', 1), ('Smith', 150), ('Baker', 144), ('Miller', 87), ('Cook', 5), ('Joyce', 1), ('Bush', 5), ('Shorter', 6), ('Klein', 1)] firstnames, weights = zip(*weighted_firstnames) wsum = sum(weights) weights_firstnames = [ x / wsum for x in weights] surnames, weights = zip(*weighted_surnames) wsum = sum(weights) weights_surnames = [ x / wsum for x in weights] weights = (weights_firstnames, weights_surnames) def synthesizer( data, weights=None, format_func=None, repeats=True): """ "data" is a tuple or list of lists or tuples containing the data. "weights" is a list or tuple of lists or tuples with the corresponding weights of the data lists or tuples. "format_func" is a reference to a function which defines how a random result of the creator function will be formated. If None,the generator "synthesize" will yield the list "res". If "repeats" is set to True, the output values yielded by "synthesize" will not be unique. """ if not repeats: memory = set() def choice(data, weights): if weights: return weighted_cartesian_choice(*zip(data, weights)) else: return cartesian_choice(*data) def synthesize(): while True: res = choice(data, weights) if not repeats: sres = str(res) while sres in memory: res = choice(data, weights) sres = str(res) memory.add(sres) if format_func: yield format_func(res) else: yield res return synthesize recruit_employee = synthesizer( (firstnames, surnames), weights = weights, format_func=lambda x: " ".join(x), repeats=False) employee = recruit_employee() for _ in range(8): print(next(employee))The above Python code returned the following:
Frank Smith Robert Smith John Baker Eve Smith John Miller Frank Miller Eve Baker Yvonne Smith
Wine Example
Let's imagine that you have to describe a dozen wines. Most probably a nice imagination for many, but I have to admit that it is not for me. The main reason is that I am not a wine drinker!
We can write a little Python program, which will use our synthesize function to create automatically "sophisticated criticisms" like this one:
This wine is light-bodied with a conveniently juicy bouquet leading to a lingering flamboyant finish!
Try to find some adverbs, like "seamlessly", "assertively", and some adjectives, like "fruity" and "refined", to describe the aroma.
If you have defined your lists, you can use the synthesize function.
Here is our solution, in case you don't want to do it on your own:
import bk_random body = ['light-bodied', 'medium-bodied', 'full-bodied'] adverbs = ['appropriately', 'assertively', 'authoritatively', 'compellingly', 'completely', 'continually', 'conveniently', 'credibly', 'distinctively', 'dramatically', 'dynamically', 'efficiently', 'energistically', 'enthusiastically', 'fungibly', 'globally', 'holisticly', 'interactively', 'intrinsically', 'monotonectally', 'objectively', 'phosfluorescently', 'proactively', 'professionally', 'progressively', 'quickly', 'rapidiously', 'seamlessly', 'synergistically', 'uniquely'] noun = ['aroma', 'bouquet', 'flavour'] aromas = ['angular', 'bright', 'lingering', 'butterscotch', 'buttery', 'chocolate', 'complex', 'earth', 'flabby', 'flamboyant', 'fleshy', 'flowers', 'food friendly', 'fruits', 'grass', 'herbs', 'jammy', 'juicy', 'mocha', 'oaked', 'refined', 'structured', 'tight', 'toast', 'toasty', 'tobacco', 'unctuous', 'unoaked', 'vanilla', 'velvetly'] example = """This wine is light-bodied with a completely buttery bouquet leading to a lingering fruity finish!""" def describe(data): body, adv, adj, noun, adj2 = data format_str = "This wine is %s with a %s %s %s\nleading to" format_str += " a lingering %s finish!" return format_str % (body, adv, adj, noun, adj2) t = bk_random.cartesian_choice(body, adverbs, aromas, noun, aromas) data = (body, adverbs, aromas, noun, aromas) synthesize = synthesizer( data, weights=None, format_func=describe, repeats=True) criticism = synthesize() for i in range(1, 13): print("{0:d}. wine:".format(i)) print(next(criticism)) print()The previous code returned the following:
1. wine: This wine is full-bodied with a intrinsically velvetly aroma leading to a lingering vanilla finish! 2. wine: This wine is light-bodied with a credibly butterscotch bouquet leading to a lingering velvetly finish! 3. wine: This wine is medium-bodied with a dynamically flabby flavour leading to a lingering food friendly finish! 4. wine: This wine is light-bodied with a interactively jammy aroma leading to a lingering flabby finish! 5. wine: This wine is light-bodied with a quickly unctuous flavour leading to a lingering unoaked finish! 6. wine: This wine is light-bodied with a dynamically refined bouquet leading to a lingering structured finish! 7. wine: This wine is medium-bodied with a quickly unoaked flavour leading to a lingering flowers finish! 8. wine: This wine is full-bodied with a credibly flowers bouquet leading to a lingering herbs finish! 9. wine: This wine is light-bodied with a compellingly juicy aroma leading to a lingering unctuous finish! 10. wine: This wine is medium-bodied with a appropriately herbs bouquet leading to a lingering lingering finish! 11. wine: This wine is light-bodied with a holisticly buttery bouquet leading to a lingering lingering finish! 12. wine: This wine is light-bodied with a enthusiastically juicy bouquet leading to a lingering tight finish!
Exercise: International Disaster Operation
It would be gorgeous, if the problem described in this exercise, would be purely synthetic, i.e. there would be no further catastophes in the world. Completely unrealistic, but a nice daydream. So, the task of this exercise is to provide synthetical test data for an international disaster operation. The countries taking part in this mission might be e.g. France, Switzerland, Germany, Canada, The Netherlands, The United States, Austria, Belgium and Luxembourg.
We want to create a file with random entries of aides. Each line should consist of:
UniqueIdentifier, FirstName, LastName, Country, Field
For example:001, Jean-Paul, Rennier, France, Medical Aid 002, Nathan, Bloomfield, Canada, Security Aid 003, Michael, Mayer, Germany, Social Worker
For practical reasons, we will reduce the countries to France, Switzerland and Germany in the following example implementation:
from bk_random import cartesian_choice, weighted_cartesian_choice countries = ["France", "Switzerland", "Germany"] w_firstnames = { "France" : [ ("Marie", 10), ("Thomas", 10), ("Camille", 10), ("Nicolas", 9), ("Léa", 10), ("Julien", 9), ("Manon", 9), ("Quentin", 9), ("Chloé", 8), ("Maxime", 9), ("Laura", 7), ("Alexandre", 6), ("Clementine", 2), ("Grégory", 2), ("Sandra", 1), ("Philippe", 1)], "Switzerland": [ ("Sarah", 10), ("Hans", 10), ("Laura", 9), ("Peter", 8), ("Mélissa", 9), ("Walter", 7), ("Océane", 7), ("Daniel", 7), ("Noémie", 6), ("Reto", 7), ("Laura", 7), ("Bruno", 6), ("Eva", 2), ("Urli", 4), ("Sandra", 1), ("Marcel", 1)], "Germany": [ ("Ursula", 10), ("Peter", 10), ("Monika", 9), ("Michael", 8), ("Brigitte", 9), ("Thomas", 7), ("Stefanie", 7), ("Andreas", 7), ("Maria", 6), ("Wolfgang", 7), ("Gabriele", 7), ("Manfred", 6), ("Nicole", 2), ("Matthias", 4), ("Christine", 1), ("Dirk", 1)] } w_surnames = { "France" : [ ("Matin", 10), ("Bernard", 10), ("Camille", 10), ("Nicolas", 9), ("Dubois", 10), ("Petit", 9), ("Durand", 8), ("Leroy", 8), ("Fournier", 7), ("Lambert", 6), ("Mercier", 5), ("Rousseau", 4), ("Mathieu", 2), ("Fontaine", 2), ("Muller", 1), ("Robin", 1)], "Switzerland": [ ("Müller", 10), ("Meier", 10), ("Schmid", 9), ("Keller", 8), ("Weber", 9), ("Huber", 7), ("Schneider", 7), ("Meyer", 7), ("Steiner", 6), ("Fischer", 7), ("Gerber", 7), ("Brunner", 6), ("Baumann", 2), ("Frei", 4), ("Zimmermann", 1), ("Moser", 1)], "Germany": [ ("Müller", 10), ("Schmidt", 10), ("Schneider", 9), ("Fischer", 8), ("Weber", 9), ("Meyer", 7), ("Wagner", 7), ("Becker", 7), ("Schulz", 6), ("Hoffmann", 7), ("Schäfer", 7), ("Koch", 6), ("Bauer", 2), ("Richter", 4), ("Klein", 2), ("Schröder", 1)] } # separate names and weights synthesize = {} identifier = 1 for country in w_firstnames: firstnames, weights = zip(*w_firstnames[country]) wsum = sum(weights) weights_firstnames = [ x / wsum for x in weights] w_firstnames[country] = [firstnames, weights_firstnames] surnames, weights = zip(*w_surnames[country]) wsum = sum(weights) weights_surnames = [ x / wsum for x in weights] w_surnames[country] = [surnames, weights_firstnames] synthesize[country] = synthesizer( (firstnames, surnames), (weights_firstnames, weights_surnames), format_func=lambda x: " ".join(x), repeats=False) nation_prob = [("Germany", 0.3), ("France", 0.3), ("Switzerland", 0.4)] profession_prob = [("Medical Aid", 0.3), ("Social Worker", 0.5), ("Security Aid", 0.2)] helpers = [] for _ in range(50): country = weighted_cartesian_choice(zip(*nation_prob)) profession = weighted_cartesian_choice(zip(*profession_prob)) country, profession = country[0], profession[0] s = synthesize[country]() uid = "{id:05d}".format(id=identifier) helpers.append((uid, country, next(s), profession )) identifier += 1 print(helpers)The above code returned the following output:
[('00001', 'France', 'Manon Nicolas', 'Social Worker'), ('00002', 'France', 'Thomas Petit', 'Medical Aid'), ('00003', 'Switzerland', 'Noémie Steiner', 'Social Worker'), ('00004', 'Switzerland', 'Daniel Müller', 'Social Worker'), ('00005', 'Germany', 'Dirk Schneider', 'Social Worker'), ('00006', 'France', 'Laura Nicolas', 'Social Worker'), ('00007', 'Germany', 'Stefanie Schmidt', 'Social Worker'), ('00008', 'France', 'Camille Nicolas', 'Social Worker'), ('00009', 'Switzerland', 'Eva Weber', 'Social Worker'), ('00010', 'Switzerland', 'Sarah Steiner', 'Security Aid'), ('00011', 'Switzerland', 'Sarah Müller', 'Social Worker'), ('00012', 'Germany', 'Ursula Klein', 'Social Worker'), ('00013', 'Switzerland', 'Walter Brunner', 'Medical Aid'), ('00014', 'Switzerland', 'Peter Zimmermann', 'Social Worker'), ('00015', 'France', 'Laura Camille', 'Security Aid'), ('00016', 'Switzerland', 'Marcel Keller', 'Medical Aid'), ('00017', 'Germany', 'Wolfgang Schneider', 'Medical Aid'), ('00018', 'Germany', 'Maria Schmidt', 'Medical Aid'), ('00019', 'Germany', 'Andreas Weber', 'Social Worker'), ('00020', 'France', 'Chloé Fournier', 'Social Worker'), ('00021', 'Switzerland', 'Hans Keller', 'Medical Aid'), ('00022', 'Germany', 'Ursula Schneider', 'Medical Aid'), ('00023', 'Switzerland', 'Urli Moser', 'Medical Aid'), ('00024', 'Germany', 'Andreas Müller', 'Medical Aid'), ('00025', 'Switzerland', 'Reto Meier', 'Medical Aid'), ('00026', 'France', 'Quentin Dubois', 'Security Aid'), ('00027', 'Germany', 'Brigitte Koch', 'Medical Aid'), ('00028', 'Germany', 'Stefanie Weber', 'Social Worker'), ('00029', 'Germany', 'Stefanie Schneider', 'Social Worker'), ('00030', 'Germany', 'Maria Müller', 'Security Aid'), ('00031', 'France', 'Alexandre Nicolas', 'Social Worker'), ('00032', 'Switzerland', 'Walter Schneider', 'Medical Aid'), ('00033', 'Germany', 'Brigitte Wagner', 'Social Worker'), ('00034', 'France', 'Laura Lambert', 'Medical Aid'), ('00035', 'Switzerland', 'Mélissa Huber', 'Social Worker'), ('00036', 'France', 'Léa Camille', 'Social Worker'), ('00037', 'Germany', 'Thomas Fischer', 'Social Worker'), ('00038', 'Germany', 'Manfred Meyer', 'Security Aid'), ('00039', 'France', 'Camille Dubois', 'Social Worker'), ('00040', 'France', 'Julien Fournier', 'Social Worker'), ('00041', 'France', 'Laura Matin', 'Security Aid'), ('00042', 'France', 'Maxime Dubois', 'Medical Aid'), ('00043', 'Switzerland', 'Daniel Frei', 'Social Worker'), ('00044', 'Germany', 'Monika Schmidt', 'Social Worker'), ('00045', 'Germany', 'Andreas Koch', 'Security Aid'), ('00046', 'France', 'Chloé Robin', 'Medical Aid'), ('00047', 'France', 'Maxime Muller', 'Social Worker'), ('00048', 'France', 'Quentin Bernard', 'Social Worker'), ('00049', 'Switzerland', 'Hans Müller', 'Medical Aid'), ('00050', 'Switzerland', 'Reto Fischer', 'Medical Aid')]
The Python code above returned the following:We received the following result: